74

Algorithms for Binary Neural Networks

For each term in Eq. 3.112, we have:

∂LS

∂k^l,i

n

= ^∂LS

∂^ˆk^l,i

n

∂^ˆk^l,i

n

∂(w^l ◦k^l,i

n ⁾

∂(w^l ◦k^l,i

n ⁾

∂k^l,i

n

= ^∂LS

∂^ˆk^l,i

n

◦1−1≤wl◦kl,i

n ^{≤1 ◦wl,}

(3.113)

∂LB

∂k^l,i

n

= λ{w^l ◦



w^l ◦k^l,i

n ^−ˆkl,i

n



+ ν[(σ^l

i^{)−2 ◦(kl}

i+ ^−μl

i+⁾

+ (σ^l

i^{)−2 ◦(kl}

i−^{+ μl}

i−^)],

(3.114)

where 1 is the indicator function that is widely used to estimate the gradient of nondiffer-

entiable parameters [199], and (σ^l

i^{)−2 is a vector whose elements are all equal to (σl}

i^)−2.

Updating w^l: Unlike the forward process, w is used in backpropagation to calculate the

gradients. This process is similar to the way to calculate ˆx from x asynchronously. Specifi-

cally, δwl is composed of the following two parts:

δwl = ^∂L

∂w^{l = ∂LS}

∂w^{l + ∂LB}

∂w^{l .}

(3.115)

For each term in Eq. 3.115, we have:

∂LS

∂w^{l =}

Il



i=1

NIl



n=1

∂LS

∂^ˆk^l,i

n

∂^ˆk^l,i

n

∂(w^l ◦k^l,i

n ⁾

∂(w^l ◦k^l,i

n ⁾

∂w^l

=

Il



i=1

NIL



n=1

∂LS

∂^ˆk^l,i

n

◦1−1≤wl◦kl,i

n ^{≤1 ◦kl,i}

n ^,

(3.116)

∂LB

∂w^{l = λ}

Il



i=1

NIl



n=1

(w^l ◦k^l,i

n ^−ˆkl,i

n ^{) ◦kl,i}

n ^.

(3.117)

Updating μ^l

i ^{and σl}

i^{: Note that we use the same μl}

i ^{and σl}

i ^{for each kernel (see Section}

3.2). So, the gradients here are scalars. The gradients δμl

i ^{and δσl}

i ^{are calculated as:}

δμl

i ^{= ∂L}

∂μ^l

i

= ^∂LB

∂μ^l

i

=

λν

C^l

i ^{×Hl×W l}

C^l

i



n=1

H^l×W ^l



p=1



(σ^l

i^)−2(μl

i ^−kl,i

n,p^),

k^l,i

n,p ^≥0,

(σ^l

i^)−2(μl

i ^{+ kl,i}

n,p^),

k^l,i

n,p ^{< 0,}

(3.118)

δσl

i ^{= ∂L}

∂σ^l

i

= ^∂LB

∂σ^l

i

=

λν

C^l

i^{×Hl×W l}

C^l

i



n=1

H^l×W ^l



p=1



−(σ^l

i^)−3(kl,i

n,p^−μl

i^)2+(σl

i^)−1,kl,i

n,p ^≥0,

−(σ^l

i^)−3(kl,i

n,p^+μl

i^)2+(σl

i^)−1,kl,i

n,p ^<0,

(3.119)

where k^l,i

n,p^{, p ∈{1, ..., Hl ×W l}, denotes the p-th element of kl,i}

n ^{. In the fine-tuning process,}

we update cm using the same strategy as center loss [245]. The update of σm,n based on

LB is straightforward and is not elaborated here for brevity.